Repository navigation
fix: run process I/O on a pool that grows to the blocking work - #114
Merged
Merged
Conversation
The client's executor was newWorkStealingPool(), a ForkJoinPool sized to availableProcessors(), but every task submitted to it blocks: two stream readers and a waitFor() per command. A ForkJoinPool only grows workers for a ManagedBlocker, which neither readLine() nor waitFor() is, so N concurrent commands need 3N blocked threads and can never have more than availableProcessors(). Starvation begins at 3N-1 > P, so on a four-core machine the second concurrent command already has a reader that is never scheduled. That command delivers no output at all, and once it has written more than a pipe buffer with nobody draining it, the child blocks in write() and stops making progress. A cached pool grows to the blocking work and reaps idle threads after 60s. It is unbounded in principle, but the bound here is the consumer's: three threads per live terraform process, which already costs far more than a thread. Threads are daemon, as ForkJoinPool's workers already were, so JVM exit behaviour is unchanged. everyConcurrentCommandDeliversItsOutput sizes itself to the machine, launching availableProcessors()/3 + 2 commands so that 3N-1 > P holds at any core count without a JVM flag or a surefire change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test class holds its own static work-stealing pool and constructs ProcessLauncher against it directly, so the client's fix does not reach it. It has the same problem for the same reason, and it is why errorStreamIsFullyCapturedWhenExitCodeFutureCompletes cannot pass at one core: that test writes 50000 lines to stderr only, and at parallelism 1 the stdout reader takes the single worker while the stderr reader never runs, so the child blocks on write. With this the suite passes at ActiveProcessorCount 1, 2 and 4. Without it, one core still fails on that test, as it does today. Three lines rather than one because Executors.defaultThreadFactory() creates non-daemon threads while ForkJoinPool's workers are daemon; keeping them daemon preserves what this pool already had. Nothing else in the class depends on work-stealing - timeoutFiresEvenWhenTheLauncherPoolHasNoSpareWorker builds its own ForkJoinPool(1) and never touches the shared pool. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
alfespa17
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Supersedes #103 by @putz612. These are the same two commits, with authorship kept, rebased onto current master. The fork belongs to an organization, so maintainers can't push to its branch.
The only conflict was at the end of
TerraformClientTest: master (#111) added the destroy/-jsontests in the same place where #103 added theawaitoverload. The resolution keeps both. Nothing else changed.Why
TerraformClientruns two blocking stream readers and a blockingwaitFor()per command onnewWorkStealingPool(). That pool is a ForkJoinPool with one worker per CPU, and it does not grow for plain blocking tasks. When3N − 1 > P(N commands, P CPUs), some reader never gets a thread, and its child blocks inwrite()once the pipe buffer fills. This PR switches to a cached pool of daemon threads, so each live process uses exactly three threads.The most likely real-world trigger is a single command on a 1-CPU executor pod that writes more than 64 KB to stderr, for example with
TF_LOG. On master that hangs. Reproduced on JDK 17, 21 and 25.Verification
All runs in Docker (
maven:3.9-eclipse-temurin-17,--init):verify: 36/36 in terraform-client, 1/1 in the sample, at default CPUs and with-XX:ActiveProcessorCount=1. Also passed at=2during review.everyConcurrentCommandDeliversItsOutput: fails. With 5 commands on 11 processors the delivered stdout counts were[20000, 20000, 20000, 20000, 0].ActiveProcessorCount=1already failsProcessLauncherTest.errorStreamIsFullyCapturedWhenExitCodeFutureCompletes. The second commit fixes that harness.Notes for the release
getExecutor()(Lombok@Getter) now returns aThreadPoolExecutorat runtime instead of aForkJoinPool. The declared type is unchanged.TerraformClient's classloader as context classloader instead of the system classloader.